apollo_dashboard: alert on committer block commit latency by itamar-starkware · Pull Request #14890 · starkware-libs/sequencer

itamar-starkware · 2026-07-28T10:46:12Z

No description provided.

reviewable-StarkWare · 2026-07-28T10:46:21Z

This change is

cursor · 2026-07-28T10:46:21Z

PR Summary

Low Risk
Dashboard and alert definition changes only; no committer runtime or data-path logic is modified.

Overview
Adds observability for committer block commit latency so slow commits show up on the committer Grafana row and trigger an alert before lag builds up.

A new Block Commit Latency panel plots p50/p95 from committer_block_commit_latency (5m rate windows). A committer_block_commit_latency_too_high alert fires when the 2-minute average commit duration exceeds 9s (~3× baseline), with a safe denominator when there are no commits. The alert is registered with other block-production-delay rules and reflected in the dev Grafana/alert JSON resources.

^{Reviewed by Cursor Bugbot for commit fcc4d0a. Bugbot is set up for automated code reviews on this repo. Configure here.}

itamar-starkware · 2026-07-28T10:46:25Z

apollo_dashboard: alert on committer block commit latency #14890 : 2 dependent PRs (#14891 , #14936 ) 👈 (View in Graphite)
apollo_committer: add per-block commit latency histogram metric #14889
apollo_committer,apollo_committer_config: warn when block commit exceeds duration threshold #14888
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

yoavGrs

@yoavGrs made 1 comment.
Reviewable status: 0 of 5 files reviewed, 1 unresolved discussion (waiting on dan-starkware and itamar-starkware).

a discussion (no related file):
@dan-starkware Do we want a new alert where the gap between consensus and batcher is > 7?

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d636e27. Configure here.}

yoavGrs

@yoavGrs made 1 comment.
Reviewable status: 0 of 5 files reviewed, 2 unresolved discussions (waiting on dan-starkware and itamar-starkware).

a discussion (no related file):
Please split this PR, one for each metric.

yoavGrs

@yoavGrs reviewed all commit messages, made 1 comment, and resolved 2 discussions.
Reviewable status: 0 of 5 files reviewed, 1 unresolved discussion (waiting on itamar-starkware).

crates/apollo_dashboard/src/alert_scenarios/block_production_delay.rs line 119 at r6 (raw file):

        format!(
            "(sum(rate({}[2m])) or vector(0)) / clamp_min(sum(rate({}[2m])) or vector(0), \
             0.0000001)",

What is the meaning of it?
Define a constant?

Code quote:

0.0000001

itamar-starkware · 2026-08-11T08:05:50Z

crates/apollo_dashboard/src/alert_scenarios/block_production_delay.rs line 119 at r6 (raw file):

Previously, yoavGrs wrote…

What is the meaning of it?
Define a constant?

It's a guard to make sure we don't divide by zero.
You can see such approach in other metrics:
cende_write_prev_height_blob_latency_too_high (alert_definitions.rs:230)

I added a constant and a comment

yoavGrs

@yoavGrs reviewed 5 files and all commit messages, and resolved 1 discussion.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on itamar-starkware).

itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 8e0f49c to ca9c6d1 Compare July 28, 2026 11:22

itamar-starkware force-pushed the committer-latency-alerts branch from 85f6cc5 to 2a70bcc Compare July 28, 2026 11:22

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread crates/apollo_dashboard/src/alert_scenarios/block_production_delay.rs Outdated

itamar-starkware marked this pull request as ready for review July 28, 2026 11:29

itamar-starkware force-pushed the committer-latency-alerts branch from 2a70bcc to fe2372b Compare July 28, 2026 12:05

cursor Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread crates/apollo_dashboard/src/panels/committer.rs Outdated

itamar-starkware force-pushed the committer-block-commit-latency-metric branch from ca9c6d1 to 64da3ff Compare July 28, 2026 13:42

itamar-starkware force-pushed the committer-latency-alerts branch 2 times, most recently from e598d8c to 57d1f68 Compare July 28, 2026 14:21

itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 64da3ff to 501db04 Compare July 28, 2026 14:21

itamar-starkware force-pushed the committer-latency-alerts branch from 57d1f68 to cee73da Compare July 29, 2026 10:28

itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 501db04 to 40ce976 Compare July 29, 2026 10:28

yoavGrs requested changes Jul 29, 2026

View reviewed changes

itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 40ce976 to 64cff66 Compare July 29, 2026 13:11

itamar-starkware force-pushed the committer-latency-alerts branch from cee73da to 3999c5f Compare July 29, 2026 13:11

itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 64cff66 to 881ce09 Compare July 30, 2026 10:38

itamar-starkware force-pushed the committer-latency-alerts branch from 3999c5f to 28b2098 Compare July 30, 2026 10:38

itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 881ce09 to 6b3efac Compare July 30, 2026 11:07

itamar-starkware force-pushed the committer-latency-alerts branch from 28b2098 to 06b11a2 Compare July 30, 2026 11:07

itamar-starkware force-pushed the committer-block-commit-latency-metric branch from 6b3efac to fd44500 Compare August 2, 2026 11:49

itamar-starkware force-pushed the committer-latency-alerts branch 2 times, most recently from 5d951af to d636e27 Compare August 2, 2026 13:23

cursor Bot reviewed Aug 2, 2026

View reviewed changes

Comment thread crates/apollo_dashboard/src/alert_scenarios/block_production_delay.rs Outdated

itamar-starkware changed the base branch from committer-block-commit-latency-metric to graphite-base/14890 August 3, 2026 06:52

itamar-starkware force-pushed the committer-latency-alerts branch from d636e27 to 1ee2983 Compare August 3, 2026 06:53

itamar-starkware force-pushed the graphite-base/14890 branch from fd44500 to 45407b1 Compare August 3, 2026 06:53

itamar-starkware changed the base branch from graphite-base/14890 to main August 3, 2026 06:53

yoavGrs requested changes Aug 3, 2026

View reviewed changes

itamar-starkware force-pushed the committer-latency-alerts branch from 1ee2983 to 29e359e Compare August 10, 2026 05:33

itamar-starkware mentioned this pull request Aug 10, 2026

apollo_dashboard: alert on committer block number lag #14936

Closed

itamar-starkware force-pushed the committer-latency-alerts branch from 29e359e to 835b2c6 Compare August 10, 2026 05:56

itamar-starkware changed the title ~~apollo_dashboard: alert on committer block number lag and block commit latency~~ apollo_dashboard: alert on committer block commit latency Aug 10, 2026

yoavGrs requested changes Aug 10, 2026

View reviewed changes

apollo_dashboard: alert on committer block commit latency

fcc4d0a

itamar-starkware force-pushed the committer-latency-alerts branch from 835b2c6 to fcc4d0a Compare August 10, 2026 13:46

itamar-starkware mentioned this pull request Aug 10, 2026

apollo_dashboard: panel for consensus vs cende state commitment infos gap #14941

Merged

yoavGrs approved these changes Aug 11, 2026

View reviewed changes

itamar-starkware added this pull request to the merge queue Aug 12, 2026

Merged via the queue into main with commit d6c11f6 Aug 12, 2026
16 checks passed

Navigation Menu

Uh oh!

apollo_dashboard: alert on committer block commit latency - #14890

apollo_dashboard: alert on committer block commit latency#14890
itamar-starkware merged 1 commit into
mainfrom
committer-latency-alerts

itamar-starkware commented Jul 28, 2026

Uh oh!

reviewable-StarkWare commented Jul 28, 2026

Uh oh!

cursor Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

itamar-starkware commented Jul 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

yoavGrs left a comment

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

yoavGrs left a comment

Uh oh!

yoavGrs left a comment

Uh oh!

itamar-starkware commented Aug 11, 2026

Uh oh!

yoavGrs left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

itamar-starkware commented Jul 28, 2026

Uh oh!

reviewable-StarkWare commented Jul 28, 2026

Uh oh!

cursor Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

itamar-starkware commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yoavGrs left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yoavGrs left a comment

Choose a reason for hiding this comment

Uh oh!

yoavGrs left a comment

Choose a reason for hiding this comment

Uh oh!

itamar-starkware commented Aug 11, 2026

Uh oh!

yoavGrs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cursor Bot commented Jul 28, 2026 •

edited

Loading

itamar-starkware commented Jul 28, 2026 •

edited

Loading